Archiving E-mails with offlineimap and evolution

I have been using Evolution for many years for my calendar and e-mail software and it has served me well. I additionally pay for e-mail hosting, following the rule that if a service is free you are most likely the "product". The internet is full of privacy invasion stories from the free services so if you want some control over your data you have to pay for hosting. The downside of this is that you don't have infinite space. I have 2GB of storage on my account, while over time the amount of e-mail I wish to keep has exceeded this.

Therefore I decided to investigate how I can archive all my e-mails, while also retaining all my e-mails in Evolution for easy access and searching.

The goals are:

• Native interface through my e-mail client (Evolution). It should look and function just like any other mail account I have.
• Identical folder structure to the original e-mail account. The archived e-mails should follow the same folder structure as the original account it was archived from.
• Immutable. When an e-mail is archived it should stay there, so for example if I delete the email from the original account it doesn't vanish from the archives
• Stored locally in a format that is open. I don't want to lose my archives due to deciding in future to switch to a new e-mail client.
• Attachments as well as e-mails are to be archived, with the original email headers intact.

I looked at lots of different software and the closest to what I wanted (that is still actively maintained) was offlineimap. This is a very powerful tool for synchonising IMAP accounts, both IMAP<->IMAP and IMAP to loads of different formats, including "maildir" formats for local access for software such as Mutt.

Maildir is also good as it is just files and folders, meaning it is an interoperable format that can be locally stored and easily backed up with my existing backup system. Evolution has native support for it (along with most open source e-mail clients) so it should show up just like any other account.

This option however was not perfect. Due to offlineimap being designed primarily for synchronisation of e-mail accounts, if I delete an e-mail on my "archive" account it deletes it from the "primary" (or vice-versa). Disabling archive changes to propagate to my "primary" can be done simply by setting the primary as "read-only" in the offlineimap config. However doing the same for primary->archive does not seem to be possible. After all offline imap is designed for synchronisation. If I set the "archive" read-only as well it won't write any e-mails at all, and if I don't then any changes to the primary will be synchronised to the archive (which is unwanted behaviour in this case).

So using offlineimap directly is not an option for me. My solution was to write a perl program that will use offlineimap to keep a local maildir synchronized, then it will use rsync to copy changed and new files to a separate "archive" maildir. As rsync does not delete files by default, this results in an immutable archive of my e-mails.

One problem I found out however is that very few people actually use offlineimap in this way. Looking online almost all the examples I saw dealt with configuring offlineimap for mutt. It seems very few wanted to use it with something like Evolution. Those few links I found about Evolution made it sound very simple. Just tell offlineimap to save your account as "maildir", add it to Evolution as a "maildir" account and job done.

The reality was a different story. Adding the account to Evolution was easy, but the account was empty. It could not see any emails, folders or not. In the end getting Evolution to see the e-mails was much harder than it should be, let alone getting my existing folder structure archived exactly as it was on my live account. It also seems there were changes to Evolution between version 2 and 3, meaning some of the recommendations were for older versions.

After much searching online I came across this debian list post which hinted at the problem. While Evolution does support maildir, the folder structure it expects is different to what offlineimap will produce. Based on what was described by the original poster I started manually moving and renaming the folders until I got the correct output for Evolution.

In a nutshell, offlineimap produces the following output for my e-mail account:

Drafts
INBOX
INBOX.Aliexpress
INBOX.defaultmail
INBOX.Lists
INBOX.Lists.DNG
INBOX.Lists.PDL
INBOX.Spam
Sent

After much trial and error, I discovered that to get the same hierarchy in my archive folder, Evolution (version 3) wants it to look like this:

cur
new
tmp
..Aliexpress
..defaultmail
.Drafts
..Lists
..Lists.DNG
..Lists.PDL
.Sent
..Spam

The Debian list post mentioned above says you can use "nametrans" rules to munge the names to something that Evolution will accept, however my reading of the documentation on nametrans gave me the impression it was too limited in its abilities (being effectively a Python lambda function call) to restructure the folders the way it is needed. The best it could do was change the folder names.
Therefore the first thing I did was to set the offlineimap separator to '.' as required by Evolution (you put "sep=." in your local account in the config file).
The moving of folders seems to be beyond offlineimaps abilities. The Debian poster used symlinks manually placed to make it work. As I was writing the wrapper program in Perl I decided to just use that instead. As such I wrote the logic in there to munge the structure to match what Evolution wants. Instead of symlinks I just moved the original Inbox files into root (otherwise I got two "Inbox" folders in Evolution, which I did not want).

Once complete the results were exactly what I wanted. Evolution shows my "Archive" account with exactly the same directory structure and all my e-mails are there, even after they were deleted from my primary account. I can happily say that all my goals have been achieved.

It was however much harder than it should have been, primarily due to lack of information online on how to do it. Hence I decided to write this article in the hope it helps others who want to do a similar thing.